# Meeting with Clint re: CPE web app
- cpeapps
- amptest log viewer
  - import progression logs
  - import part names
- jira scraper
- 

## Existing log viewer

https://stackoverflow.com/questions/10525725/which-nosql-database-should-i-use-for-logging

MongoDB Capped Collections are extremely popular and suitable for logging, with the added bonus of being 'schema less', which is usually a semantic fit for logging. Often we only know what we want to log well into a project, or after certain issues have been found in production. Relational databases or strict schemas tend to be difficult to change in these cases, and attempts to make them 'flexible' tends just to make them 'slow' and difficult to use or understand.


AmptestLogIndex
  https://github.com/Ampcontrol-CPE/AmptestLogTool/blob/main/AmptestLogIndex/management/commands/populateparts.py
  MongoDB "collection" is... a table?

  parser = AmptestLogParserWeb.argParserBuilder()  # buils arguments
  AmptestLogParserWeb.parseToMongo(args)  # parse logs
  command = 'rsync -azP "/media/share/Production/Test Programs/Amptest Logs/" "/home/ubuntu/Amptest-Web-App/Amptest Logs"'
              # command = 'robocopy "G:\Production\Test Programs\Amptest Logs" "C:\Amptest Logs" /NOCOPY /E /copy:DAT /Z /MT:32 /log+:amptestlogcopy.log"'

/Volumes/Groups/Production/Test Programs/Amptest Logs

Django, JSONField and sqlite: supported as of Python 3.9
    https://code.djangoproject.com/wiki/JSON1Extension
    https://docs.djangoproject.com/en/5.0/ref/models/fields/#django.db.models.JSONField
    https://docs.djangoproject.com/en/5.0/topics/db/queries/#querying-jsonfield


log_import: import_time, file_modification_time, file_hash
log_sequence: log_import
log_line: import_id

    id = models.ObjectIdField(db_column="_id")
    File_Name = models.CharField(max_length=256)
    Date = models.CharField(max_length=128)
    Part_Number = models.IntegerField()
    Serial_Number = models.BigIntegerField()
    Test_Name = models.TextField()
    Script_Name = models.TextField()
    Script_Version = models.CharField(max_length=128)
    Fixate_Version = models.CharField(max_length=128)
    Report_Format = models.IntegerField()
    Index_String = models.TextField()
    Driver_Serials = models.TextField()
    Result = models.CharField(max_length=128)
    Runtime = models.FloatField()
    Runtime_WO_User = models.FloatField()
    Repeat_Tests = models.BooleanField()
    Full_Path = models.CharField(max_length=512)
    RAW = models.TextField()


Update 6th January 2022: sqlite-utils 3.20 introduced a new sqlite-utils insert ... --lines option for importing raw lines, so you can now achieve this without using jq at all. See Inserting unstructured data with --lines and --text for details.

Now I had a SQLite table with a single column, line. Next step: parse that nasty log format.

https://www.debuggex.com/

Now I had a SQLite table with a single column, line. Next step: parse that nasty log format.

To my surprise I couldn’t find an existing Python library for parsing key=value key2="quoted value" log lines. Instead I had to figure out a regular expression:

([^\s=]+)=(?:"(.*?)"|(\S+))

https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-convert

sqlite-utils insert logs.db loglines access.log --convert '
type, source, _, verb, path, _, status, _ = line.split()
return {
    "type": type,
    "source": source,
    "verb": verb,
    "path": path,
    "status": status,
}' --lines

https://ampcontrol.atlassian.net/wiki/spaces/CPEK/pages/121176103/Log+Format

idea... save raw line into json array, then migrate it to proper format